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1.0  Summary 

This  project  was  conducted  within  the  facilities  at  71 1  HPW/RHC  under  contract  FA8650- 
06-C-6755;  program  manager  Dr.  Paul  Havig,  711  HPW/RHCP,  937-255-8737.  Contractors 
participating  in  the  data  collection  are  University  of  Dayton  Research  Institute  (UDRI) 
employees.  Personnel  from  RHC  and  UDRI  interacted  with  the  subjects  by  providing 
training  for  C2  task,  administering  the  test,  and  conducting  outbrief  tasks  once  the  test 
was  completed. 

This  effort  involved  two  specific  Targeted  Investigations  explored  the  use  of  large-screen 
displays  paired  with  desktop  displays,  typical  in  AOCs  and  other  C2  environments. 
Specifically,  two  research  areas  were  outlined:  1)  the  use  of  prompting  to  direct  the 
operator’s  attention  to  either  the  large-screen  display  or  the  small-screen  display;  and,  2) 
communication  fonnats  which  could  be  used  by  an  operator  to  collaborate  with  a  second 
operator.  The  first  research  area  evaluated  the  perfonnance  of  participants  in  a  Command 
and  Control  (C2)  environment  using  auditory  and  visual  stimuli  as  prompting  cues  to 
switch  attention  between  a  small  screen  and  large  screen  display.  Ten  participants 
completed  eight  monitoring  task  sessions  consisting  of  four  different  conditions;  no 
cueing,  auditory  cueing,  visual  cueing,  and  combined  auditory  and  visual  cueing. 

Prompting,  in  any  form,  improved  response  times  as  compared  to  no  prompt.  No 
differences  were  found  among  prompt  type.  The  second  research  area  focused  on  three 
communication  mediums  which  could  be  implemented  in  the  C2  setting:  spoken  word, 
handwritten,  and  chat.  Participants  monitored  a  large  display  depicting  the  flights  of 
three  UAVs,  while  simultaneously  tracking,  on  a  separate  smaller  screen,  the  status  of 
critical  performance  parameters  of  the  UAVs.  Results  indicate  a  time  and  accuracy 
advantage  for  spoken  word,  with  readback  providing  a  speed-accuracy  tradeoff. 

Participant  preference  was  evenly  divided  in  terms  of  communication  technique  and 
presence  of  readback. 


2.0  Introduction 
2.1  Scope 

The  scope  of  this  effort  was  to  address  fundamental  human  factors  issues  associated  with 
large  shared  displays  and  other  collaboration  technologies  currently  in  use  or  proposed 
for  use  in  collaborative  command  and  control  environments,  such  as  Air  Operations 
Centers  (AOCs). 


2.2  Background 

The  increasing  pervasiveness  of  hardware  and  software  systems  has  resulted  in  reduced 
costs  associated  with  technologies  for  infonnation  display  and  sharing.  While  these 
savings  allow  more  and  more  organizations  to  purchase  advanced  technologies,  often 
acquisition  and  implementation  decisions  are  made  without  full  understanding  of  the 
potential  benefits  the  technology  offers.  Although  suites  of  collaborative  technologies 
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have  been  acquired  and  integrated  in  many  C2  environments,  reports  from  the  operational 
community  are  mixed  with  regard  to  the  usability,  usefulness,  and  effectiveness  of  these 
technologies. 

The  quantity  of  infonnation  available  to  operators  today  and  infonnation  flow  planned 
for  network-centric  warfare  is  staggering.  Care  must  be  taken  to  avoid  unintended 
consequences  such  as  information  overload,  attentional  capture,  change  blindness,  and 
decision  biases  associated  with  certain  modes  of  information  presentation.  Solutions  must 
consider  improving  shared  situation  awareness  (SA)  and  increased  speed  of  command. 
Effective  use  of  collaborative  technologies  requires  investigation  of  these  types  of 
fundamental  human  factors  issues  associated  with  their  utility. 


2.3  Tasks 

The  approach  outlined  for  this  effort  involved  two  specific  Targeted  Investigations  to 
address  some  of  the  issues  above.  Each  Targeted  Investigation  explored  the  use  of  large- 
screen  displays  paired  with  desktop  displays,  typical  in  AOCs  and  other  C2 
environments.  Specifically,  two  research  areas  were  outlined:  1)  the  use  of  prompting  to 
direct  the  operator’s  attention  to  either  the  large-screen  display  or  the  small-screen 
display;  and,  2)  communication  fonnats  which  could  be  used  by  an  operator  to 
collaborate  with  a  second  operator. 

Targeted  Investigation  1,  Maintaining  Vigilance  with  Auditory  and  Visual  Cues  in 
Command  and  Control  Environments,  evaluated  the  performance  of  participants  in  a 
Command  and  Control  (C2)  environment  using  auditory  and  visual  stimuli  as  prompting 
cues  to  switch  attention  between  a  small  screen  and  large  screen  display.  The  use  of 
spatial  audio  displays  has  been  shown  to  reduce  workload  and  improve  target  detection 
times1.  The  current  design  employed  a  two-screen  model  with  multiple  targets  on  each  as 
well  as  a  multimodal  cueing  strategy.  Ten  participants  completed  eight  monitoring  task 
sessions  consisting  of  four  different  conditions;  no  cueing,  auditory  cueing,  visual  cueing, 
and  combined  auditory  and  visual  cueing.  Participants  monitored  a  large  display 
depicting  the  flights  of  three  UAVs,  while  simultaneously  tracking,  on  a  separate  smaller 
screen,  the  status  of  critical  performance  parameters  of  the  UAVs.  Reaction  times  and 
accuracy  rates  as  well  as  perceived  workload  were  compared  across  all  four  conditions. 

It  was  found  that  prompting,  in  any  fonn,  will  improve  response  times  as  compared  to  no 
prompt.  No  differences  were  found  among  prompt  type.  The  findings  of  this  study  apply 
to  the  C2  environment,  and  may  also  apply  to  other  multi-task  environments  requiring 
monitoring  of  multiple  visual  displays. 

Targeted  Investigation  2,  The  Effect  of  Communication  Technique  and  Presence  of 
Readback  in  Command  and  Control  Environments ,  focused  on  three  communication 
mediums  which  could  be  implemented  in  the  C2  setting.  Efficient  and  effective 
communication  between  operators  in  a  C2  environment  is  essential  to  mission  success. 
Modem  technology  affords  several  communication  options.  The  second  Targeted 
Investigation  evaluated  three  of  these  options:  spoken  word,  handwritten,  and  electronic 
chat.  Participants  monitored  a  large  display  depicting  the  flights  of  three  UAVs,  while 
simultaneously  tracking,  on  a  separate  smaller  screen,  the  status  of  critical  perfonnance 
parameters  of  the  UAVs.  Using  one  of  the  three  methods  participants  communicated  key 
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events  to  a  secondary  operator.  For  half  the  trials,  the  secondary  operator  provided 
feedback  (confirmation  or  request  for  clarification);  for  the  other  half,  no  readback  was 
given.  Performance  times,  accuracy,  and  subjective  workload  data  was  collected.  User 
preference  and  familiarity  with  each  form  of  communication  was  also  evaluated.  Initial 
results  indicate  a  time  and  accuracy  advantage  for  spoken  word,  with  readback  providing 
a  speed-accuracy  tradeoff.  Participant  preference  was  evenly  divided  in  terms  of 
communication  technique  and  presence  of  readback. 

Below  is  a  detailed  description  of  each  Targeted  Investigation. 


3.0  Targeted  Investigation  1:  Maintaining  Vigilance  with  Auditory  and 
Visual  Cues  in  Command  and  Control  Environments. 

3.1  The  C2  Environment  and  Divided  Attention 

The  tenn  “Command  and  Control”  (C2)  can  be  used  to  describe  a  variety  of  applications 
requiring  a  centralized  command  center  or  facility.  Typically,  however,  configuration  of 
this  facility  involves  a  large  screen  display,  or  data  wall,  visible  to  multiple  operators. 
Simultaneously,  a  given  operator  also  has  his  or  her  own  dedicated  workstation.  This 
setup,  by  design,  will  likely  lead  to  tasks  that  require  the  operator’s  attention  at  the  data 
wall,  and  other  tasks  that  will  require  the  operator’s  attention  to  his  or  her  dedicated 
workstation.  Divided  attention  in  such  a  multitask  environment  can  potentially  lead  to 
operator  perfonnance  decrements  with  regard  to  one  or  both  tasks. 

Such  decrements,  in  theory,  could  be  mitigated  by  directing  the  operator’s  attention, 
when  appropriate,  to  the  appropriate  display.  Gunn  et  al."  discuss  the  advantages  of 
sensory  cueing  strategies;  that  is,  those  which  utilize  a  simple  physical  change  to  the 
stimuli  in  order  to  attract  the  operator’s  attention.  Use  of  such  cueing  strategies  has  a 
demonstrated  advantage  over  no  cueing  whatsoever,  as  well  as  cognitive  cueing 
strategies,  which  require  symbolic  interpretations  and  manipulations  (e.g.,  determining 
the  sum  of  a  number  array  before  responding). 

The  use  of  spatial  audio  displays  has  been  shown  to  reduce  workload  and  improve  target 
detection  times3.  Traditionally,  such  a  paradigm  utilizes  a  single  display,  containing 
single  or  multiple  targets.  The  implications  of  such  findings  generally  involve  display 
design  for  combat  vehicles,  such  as  aircraft  or  tanks.  The  strength  of  such  displays  is 
their  non-intrusive  nature,  which  take  advantage  of  a  relatively  free  sensory  channel 
(auditory)  in  order  to  convey  information  to  the  visually  tasked  operator.  Such  cross- 
modal  time  sharing  can  lead  to  improved  perfonnance4,  especially  if  the  tasks  in  question 
are  of  a  highly  visual  in  nature5.  In  the  C2  environment,  when  directing  an  operator’s 
visual  attention  to  the  common  or  individual’s  display,  an  auditory  cue  might  benefit  the 
operator  similarly.  Basic  research  has  shown  an  advantage  in  response  times  to  auditory 
stimuli,  as  compared  to  visual  sitimuli6. 

The  objective  of  Targeted  Investigation  1  was  to  evaluate  the  performance  of  an 
individual  in  a  command  and  control  (C2)  environment,  in  which  they  were  required  to 
switch  attention  between  a  large  screen  display  and  a  dedicated  workstation.  The  large 
screen  display  contained  general  infonnation  depicting  the  flights  of  three  UAVs,  and  the 
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dedicated  workstation  contained  infonnation  specific  to  the  participant  depicting  critical 
performance  parameters  of  each  UAV.  The  study  compared  various  means  to  cue  the 
user  to  direct  attention  between  the  two  displays.  Applications  of  the  study  involved  the 
Command  and  Control  (C2)  environment,  although  the  results  may  also  be  applied  to 
combat  vehicle  display  design. 


3.2  Method 

3.2.1  Participants 

Ten  adult  participants  (50  percent  female)  were  sampled  from  the  University  of  Dayton 
Research  Institute  and  the  Air  Force  Research  Laboratory  at  Wright  Patterson  Air  Force 
Base.  All  participation  was  voluntary.  All  participants  had  nonnal  or  corrected-to-normal 
vision. 

3.2.2  Materials 

3.2.2. 1  Task  Environment.  A  simulated  task  environment  was  created  with  a  large, 
common  display  and  one  smaller  workstation.  A  task  set,  representative  of  a  C2 
environment  task,  was  developed  requiring  the  participant  to  interact  with  the  large 
screen  display  (LSD)  and  a  dedicated  small  screen  display  (SSD).  The  C2  task 
environment  was  created  using  the  Distributed  Dynamic  Decision-Making  (DDD) 
software  tool6.  The  task  involved  monitoring  a  simulation  of  three  unmanned  air 
vehicles  (UAVs),  with  route  and  position  depicted  on  the  LSD,  and  gauges  representing 
the  status  of  specific  aircraft  infonnation  contained  on  the  SSD. 

3. 2.2. 2  Room.  A  10’xl2’  laboratory  on  the  campus  of  Wright  Patterson  Air  Force  base 
was  used  to  conduct  the  study.  The  LSD  was  affixed  to  the  back  wall  of  the  laboratory 
and  the  SSD  sat  on  a  table  approximately  6  feet  in  front  of  the  LSD. 

The  SSD  was  a  17”  desktop  monitor  supported  by  a  MIcit  Desktop  running  on  Windows 
XP  Professional  which  displayed  the  lower  monitoring  task  as  well  as  the  visual  cues  (see 
Figure  1). 
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Figure  1:  Small  screen  display;  Study  1 


A  wall  mounted  (80”  horizontal,  1280  x  1024  resolution)  projection  screen  served  as  the 
LSD.  Image  was  projected  using  an  InFocus©  projector  from  a  Dell  Dimension  Desktop 
XPS  running  on  Windows  XP  Professional.  Figure  2  depicts  the  LSD. 


Figure  2:  Large  screen  display;  Study  1 


3. 2. 2. 3  Auditory  Cues.  Auditory  tones  played  via  Logitech  stereo  headphones  cued  an 
event.  The  tone  ascended  to  indicate  a  LSD  event  and  descended  to  indicate  a  SSD 
event.  The  tone  played  monaurally  to  the  left  ear  to  indicate  an  event  on  the  left  of  the 
screen,  monaurally  to  the  right  ear  to  indicate  an  event  on  the  right  of  the  screen,  or 
binaurally  to  indicate  an  event  on  the  middle  of  the  appropriate  screen. 

3. 2. 2.4  Visual  Cues.  Visual  cues  of  an  event  involved  presentation  of  a  red  arrow  (see 
Figure  1)  via  a  display  bar  at  the  top  of  the  SSD.  Directionality  was  achieved  by 
presenting  the  arrow  in  one  of  three  positions  within  the  display  bar:  left,  center,  or  right 
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to  indicate  event  location  respectively.  The  cue  pointed  up  to  indicate  an  event  on  the 
LSD,  or  down  to  indicate  an  event  on  the  SSD. 

3.2. 2. 5  Subjective  Measures.  The  researchers  created  a  brief  post  procedure  survey  that 
included  items  that  addressed  the  respondents’  perceived  performance  for  each  type  of 
event  notification  and  preferred  method  of  event  notification.  The  NASA-TLX  was 
administered  electronically  between  each  experimental  session  to  collect  participants’ 
perceived  workload. 

3.2.3  Design 

Participants  completed  one  practice  monitoring  session  and  eight  monitoring  sessions 
which  each  lasted  approximately  six  minutes  in  length.  Depending  on  the  condition  a 
participant  was  either  cued  by  auditory  tones,  visual  signals,  both  or  not  at  all.  However, 
not  every  event  within  each  condition  was  cued.  This  was  included  in  the  procedure  to 
create  the  potential  to  test  for  participants’  complacency  with  the  task.  Timing  data  was 
collected  for  each  event  as  well  as  accuracy  of  response  (correct,  incorrect),  cue  presence 
(present,  not  present)  and  cue  type  (audio,  visual,  none,  both). 

3.2.4  Procedure 

Each  participant’s  session  lasted  approximately  two  hours  including  training.  When  they 
arrived,  participants  signed  an  infonned  consent  and  were  verbally  briefed  on  the  task 
instructions. 

Participants  had  two  tasks  which  were  to  be  performed  simultaneously.  The  first  task 
was  to  monitor  a  simulation  of  three  UAV’s  represented  on  the  LSD,  and  indicate  when 
they  encountered  an  event  (artillery,  chemBio,  bunker).  An  event  was  indicated  by  a 
labeled  icon  which  appeared  on  the  LSD.  Participants  responded  to  the  presence  of  an 
event  by  selecting  the  corresponding  radio  button  on  the  SSD  via  the  computers  mouse. 

The  participants’  second  task  was  to  monitor  three  gauges  which  indicated  the  status  of 
three  critical  parameters  (i.e.,  fuel  remaining,  signal  strength,  engine  temperature)  of  the 
UAV’s.  When  the  critical  parameters  exceeded  their  optimal  perfonnance  range,  the 
gauge  moved  from  a  green  area  into  a  red  area.  Participants  responded  to  the  critical 
parameters  by  clicking  on  the  indicator  button  on  the  SSD  via  the  computers  mouse  when 
the  indicator  exceeded  its  limits. 

After  completing  all  eight  sessions,  participants  were  debriefed  and  asked  to  complete  a 
post  experiment  survey. 


3.3  Results 

3.3.1  Reaction  Time 

A  2-way  repeated-measures  ANOVA  was  conducted  to  examine  the  effect  of  cue  type 
and  trial  period  on  reaction  time  to  an  on-screen  UAV  incident.  The  within-subjects 
variables  were  cue  type,  with  four  levels  (auditory  cues,  visual  cues,  simultaneous 
auditory  and  visual  cues,  and  no  cue),  and  two  trial  periods  for  each  cue  type.  The  results 
of  the  ANOVA  are  presented  in  Table  1,  while  Table  2  presents  the  means  and  standard 
deviations  for  these  main  effect  groups.  Alpha  level  .05  was  used. 
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Table  1 :  ANOVA  Summary  Table  for  Reaction  Time 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Stimulus 

Sphericity  Assumed 

3793557.312 

3 

1264519.104 

9.077 

.000 

Greenhouse-Geisser 

3793557.312 

1.693 

2241308.565 

9.077 

.004 

Huynh-Feldt 

3793557.312 

2.043 

1857286.256 

9.077 

.002 

Lower-bound 

3793557.312 

1.000 

3793557.312 

9.077 

.015 

Error(Stimulus) 

Sphericity  Assumed 

3761274.904 

27 

139306.478 

Greenhouse-Geisser 

3761274.904 

15.233 

246915.053 

Huynh-Feldt 

3761274.904 

18.383 

204609.014 

Lower-bound 

3761274.904 

9.000 

417919.434 

Trial 

Sphericity  Assumed 

28875.169 

1 

28875.169 

.119 

.738 

Greenhouse-Geisser 

28875.169 

1.000 

28875.169 

.119 

.738 

Huynh-Feldt 

28875.169 

1.000 

28875.169 

.119 

.738 

Lower-bound 

28875.169 

1.000 

28875.169 

.119 

.738 

Error(Trial) 

Sphericity  Assumed 

2177001.263 

9 

241889.029 

Greenhouse-Geisser 

2177001.263 

9.000 

241889.029 

Huynh-Feldt 

2177001.263 

9.000 

241889.029 

Lower-bound 

2177001.263 

9.000 

241889.029 

Stimulus ’Trial 

Sphericity  Assumed 

372156.613 

3 

124052.204 

1.037 

.392 

Greenhouse-Geisser 

372156.613 

2.334 

159428.945 

1.037 

.382 

Huynh-Feldt 

372156.613 

3.000 

124052.204 

1.037 

.392 

Lower-bound 

372156.613 

1.000 

372156.613 

1.037 

.335 

ErrorfStimulusTrial) 

Sphericity  Assumed 

3231374.727 

27 

119680.545 

Greenhouse-Geisser 

3231374.727 

21.009 

153810.593 

Huynh-Feldt 

3231374.727 

27.000 

119680.545 

Lower-bound 

3231374.727 

9.000 

359041.636 

Due  to  a  significant  violation  of  the  assumption  of  sphericity  (/?=.04 1 ),  the  Greenhouse- 
Geiser  test  was  used  to  adjust  the  degrees  of  freedom.  A  significant  main  effect  for  cue 
type  was  found,  F(1.693,  15.233)  =  9.077,/?  <  0.01.  Participants  responded  significantly 
slower  in  the  “no  cue”  condition  than  any  other  condition.  There  were  no  significant 
differences  in  reaction  time  between  the  other  stimulus  conditions.  Additionally,  there 
was  not  a  significant  main  effect  of  trial  period,  F(l,  9)  =  .1 19,/?  =  .738,  nor  the 
interaction  of  cue  type  and  trial  period,  F(2.334,  21.009)  =  1.037,/?  =  .382. 


Table  2:  Response  Times  in  Seconds 


Factor 

M 

SD 

Auditory 

26.359 

3.798 

Visual 

27.837 

4.719 

Combined 

27.487 

4.864 

No  Cue 

32.096 

7.303 

Trial  1 

28.635 

5.408 

Trial  2 

28.255 

5.982 
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Each  cued  condition  contained  both  cued  and  non-cued  events  to  ensure  participant 
vigilance.  Therefore,  a  separate  analysis  was  conducted  using  only  data  collected  from 
each  of  the  “true”  (cued)  events  (i.e.,  non-cued  events  in  the  cued  condition  were 
excluded  from  the  analysis). 

3. 3. 1.1  “True”  Only  Events.  A  2-way  repeated-measures  ANOVA  examining  the  effect 
of  “true”  cues  and  trial  period  on  reaction  time  was  conducted.  Table  3  presents  the 
means  and  standard  deviations  for  the  main  effect  groups.  Alpha  level  .05  was  used.  A 
significant  main  effect  for  cue  type  was  found,  F(3,  27)  =  13.577,  p  <  0.001.  Participants 
responded  significantly  slower  in  the  “no  cue”  condition  than  any  other  condition.  There 
was  not  a  significant  main  effect  of  trial  period,  F(l,  9)  =  .363,  p  =  .562,  nor  the 
interaction  of  cue  type  and  trial  period,  F(3,  27)  =  1.61 1,  p  =  .210. 


Table  3:  “True”  only  reaction  time  means  and  standard  deviations 


Factor 

M 

SD 

Auditory 

25.193 

4.571 

Visual 

26.733 

4.883 

Combined 

26.105 

4.560 

No  Cue 

32.096 

7.303 

Trial  1 

27.727 

5.937 

Trial  2 

27.094 

6.172 

3.3.2  Workload 

A  2-way  repeated-measures  ANOVA  was  conducted  to  examine  the  effect  of  cue  type 
and  trial  period  on  subjective  workload.  Table  4  presents  the  means  and  standard 
deviations  for  these  main  effect  groups.  Alpha  level  .05  was  used.  Due  to  a  significant 
violation  of  the  assumption  of  sphericity  (p=. 034),  the  Greenhouse-Geiser  test  was  used 
to  adjust  the  degrees  of  freedom.  There  was  no  significant  main  effect  for  cue  type, 

F(  1.625,  14.626)  =  A05,p  =  .633,  trial  period,  F(  1,  9)  =  1 .688,  =  .226,  or  the 
interaction  of  cue  type  and  trial  period,  F( 2.285,  20.562)  =  .392 ,p  =  .760. 
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Table  4:  Workload  means  and  standard  deviations 


Factor 

M 

SD 

Auditory 

46.533 

20.837 

Visual 

46.467 

19.660 

Combined 

49.683 

20.930 

No  Cue 

47.067 

21.564 

Trial  1 

48.300 

22.734 

Trial  2 

46.575 

20.325 

3.3.3  Rank  Preference  and  Perceived  Performance 

A  within-subjects  comparison  of  rank  orders  was  perfonned  on  the  cue  type  using  the 
Friedman  test  on  both  preference  and  perceived  performance.  Alpha  level  .05  was  used. 
The  Friedman  test  significantly  placed  the  cue  types  in  order  of  preference  by  Combined 
(1.7),  Visual  (2.2),  Auditory  (2.3),  and  None  (3.8)  (Friedman  X2  =  14.76,/?  =  .002). 
Additionally,  the  Friedman  test  significantly  placed  the  cue  types  in  order  of  perceived 
performance  by  Combined  (2.0),  Auditory  (2.1),  Visual  (2.2),  and  None  (3.7)  (Friedman 
X2  =  1 1.64,/?  =  .009).  Multiple  comparisons  revealed  that  No  Cue  was  preferred 
significantly  less  than  Audio  (/?  =  .012),  Visual  (/?  =  .004),  and  Both  (p  =  .010).  Also, 
multiple  comparisons  revealed  that  performance  with  no  prompt  was  perceived  as  less 
than  Audio  (p  =  .024),  Visual  (p  =  .004),  and  Both  (p  =  .017).  Multiple  comparisons 
revealed  no  other  significant  differences  in  rankings  in  either  preference  or  liking. 

3.3.4  Errors 

No  statistical  analysis  was  conducted  on  error  data,  as  participants  across  all  four 
conditions  did  not  make  enough  errors  for  an  accurate  analysis. 


3.4  Discussion 

The  general  result  that  any  type  of  prompt  is  better  than  no  prompt  was  expected.  This 
pattern  held  true  in  terms  of  performance  times,  perceived  perfonnance,  and  preference. 
However,  the  lack  of  statistical  evidence  favoring  one  prompting  strategy  over  another 
was  unexpected,  especially  in  light  of  other  research  in  this  area6.  This  lack  of  evidence 
held  true  across  all  measures,  including  response  time  and  preference.  Several  reasons 
may  account  for  this  non-finding  and  are  discussed  below. 

Participant  preference  indicated  that  some  type  of  prompting  was  preferred  over  no 
prompting.  However,  no  specific  prompt  strategy  was  preferred.  This  may  be  due  to 

o 

differing  learning  styles  among  the  participant  pool  .  A  standard  learning  style  survey 
was  not  implemented  as  part  of  Targeted  Investigation  1;  however,  conflicting  post  test 
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feedback  regarding  preferred  prompt  technique  provides  some  anecdotal  evidence.  For 
example,  a  majority  of  participants  preferred  the  combination  of  audio  and  visual 
prompts.  The  reason  given  for  this  preference  was  to  receive  “as  much  prompting  as 
possible.”  However,  to  others,  the  combination  of  audio  and  visual  prompting  was 
excessive.  Two  participants  stated  that  they  preferred  audio,  as  it  did  not  interfere  with 
the  visual  search  task.  An  additional  participant  preferred  visual  prompting  only;  the 
reason  given  that  the  auditory  prompt,  which  resembled  a  siren,  increased  the 
participant’s  stress  level.  With  such  a  disparity  of  reasons  given  for  preference  ranking, 
future  research  should  involve  implementing  a  learning  style  test,  such  as  the  VARK9. 

Another  non-significant  finding  was  that  auditory  prompting  showed  no  advantage  over 
other  prompt  formats.  The  intent  of  using  auditory  prompting  during  a  visual  task  is  to 
take  advantage  of  a  second,  unused  perceptual  channel.  While  this  finding  may  initially 
seem  surprising,  Wickens  states  that,  if  the  physical  distance  between  the  competing 
visual  elements  (e.g.,  visual  task  and  visual  prompt)  is  not  great  enough,  advantage  of 
auditory  is  not  realized.  From  the  participant’s  point  of  view,  the  visual  prompt  was 
strategically  positioned  to  minimize  visual  orienting.  An  unintended  consequence  of  this 
strategy  may  have  been  to  minimize  the  need  for  a  secondary  channel  (e.g.,  audio)  in 
terms  of  prompting. 

Finally,  it  is  recommended  that  future  studies  increase  the  vigilance  requirement 
associated  with  the  experimental  task.  The  sessions  in  Targeted  Investigation  1  lasted 
approximately  18  minutes,  with  an  even  occurring  every  20  seconds.  While  this 
arrangement  generated  a  large  number  of  data  points  in  a  short  time  span,  the  need  for  a 
longer  period  of  data  collection  may  exist.  The  non-significant  analysis  of  workload 
scores  may  have  been  due  to  a  lack  of  vigilance  required  for  each  session.  Longer  data 
collection  sessions  may  also  help  to  illustrate  potential  long-tenn  benefits  of  a  given 
prompt  strategy  over  another.  In  general,  future  research  could  space  events  more 
randomly  over  a  larger  period  of  time. 

The  results  of  Targeted  Investigation  1  indicate  that  the  provision  of  some  form  of 
prompting  will  benefit  the  participant.  However,  objective  and  subjective  results  to  not 
point  to  a  specific  format  as  preferred.  In  the  meantime,  designers  should  consider 
multiple  formats  to  allow  for  individual  user  preferences.  Additionally,  future  research 
should  include  longer  test  sessions,  to  create  a  true  vigilance-oriented  task.  Also,  the 
impact  of  participant  learning  style  on  prompt  preference  should  be  investigated,  to  more 
effectively  detennine  the  need  for  multiple  prompting  strategies. 


4.0  Targeted  Investigation  2:  The  Effect  of  Communication  Technique  and 
Presence  of  Readback  in  Command  and  Control  Environments 

4.1  The  C2  Environment  and  Team  Communication 

“Command  and  Control”  (C2)  facilities  are  typically  configured  with  a  large-screen 
display,  or  data  wall,  visible  to  multiple  operators.  Simultaneously,  a  given  operator  also 
has  his  or  her  own  dedicated  workstation  for  work  activities  specific  to  that  user.  In  such 
a  setting,  communication  between  operators  may  take  several  forms,  the  basis  for  which 
may  involve  technology,  personal  preference,  or  both.  The  most  effective  form  of 
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communication,  however,  may  not  be  the  most  preferred.  Nielson  and  Mack10  indicate 
that  users,  even  when  given  proof  that  their  performance  is  better  when  using  a  given 
technology,  will  not  use  that  technology  if  they  do  not  prefer  it. 

Another  issue  in  detennining  an  effective  communication  method  is  that  of  fostering 
team  cohesiveness.  For  example,  prior  studies  indicate  that  trust  is  generally  slow  to 
develop  and  particularly  fragile  when  team  members  do  not  get  an  opportunity  to  interact 
face-to-face10'11.  Therefore,  the  selection  and  implementation  of  a  communication 
technique  must  balance  performance  and  user  acceptance,  ideally  retaining  some  factors 
associated  with  face-to-face  interactions. 

Three  communication  techniques  with  potential  to  be  utilized  in  the  C2  setting  are 
spoken  word,  written  messages,  and  the  use  of  electronic  chat  software.  Each  technique 
offers  advantages  and  disadvantages.  For  example,  while  spoken  word  is  likely  to  be 
most  familiar  to  the  greatest  number  of  users,  it  can  be  intrusive  to  other  operators  in  the 
task  environment.  As  well,  it  appears  that  documentation  or  capture  of  spoken 
communication  can  be  cumbersome.  Handwritten  messages  and  chat  are  both  visual 
mediums,  providing  a  smoother  transcription  process  than  spoken  word.  This  is 
especially  true  in  the  case  of  chat,  where  an  electronic  text  file  can  be  generated  literally 
as  a  session  progresses.  However,  some  users  may  not  be  comfortable  with  the  novelty 
of  chat  software.  Other  tradeoffs  apply  to  handwritten  communication,  such  as  legibility 
and  the  logistics  of  delivery. 

Because  a  collaborative  environment  involves  at  least  two  individuals,  the  behavior  of 
the  second  individual  can  directly  impact  the  overall  quality  of  communication.  The 
concept  of  read-back,  or  offering  a  confirmation  or  request  for  clarification,  is  standard 
procedure  in  settings  such  as  aviation13  and  medical  situations14.  Thus,  a  second  factor 
that  may  influence  the  effectiveness  of  a  given  communication  technique  is  the  presence 
or  absence  of  return  communication  from  the  second  individual. 

Given  the  array  of  capabilities  and  options  provided  by  today’s  technology,  a 
multifaceted  investigation  was  developed  to  evaluate  spoken  word,  written  messages,  and 
the  use  of  chat  software  as  communication  techniques  in  the  C2  setting.  The  impact  of 
“read-back”  was  also  evaluated.  The  intent  of  this  variable  was  to  not  necessarily 
evaluate  whether  performance  increased,  but  rather  subjective  preference  associated  with 
perceived  read-back  perfonnance  for  a  given  communication  technique. 


4.2  Method 
4.2.1  Design 

The  experimental  design  was  a  2x3  within  subjects  design.  The  two  independent 
variables  were  Readback  (Present,  Absent),  and  Communication  Type  (Chat,  Spoken 
Word,  Written). 
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4.2.2  Participants 

Ten  adult  participants  (50  percent  female)  were  sampled  from  the  University  of  Dayton 
Research  Institute  and  the  Air  Force  Research  Laboratory  at  Wright  Patterson  Air  Force 
Base.  All  participation  was  voluntary.  All  participants  had  nonnal  or  corrected-to- 
normal  vision. 

4.2.3  Materials 

4.2.3. 1  Task  Environment.  A  simulated  C2  setting  was  developed  for  this  study.  This 
setting  included  a  large,  common  display  and  one  smaller  workstation  for  individual  use. 
The  task  set  required  the  participant  to  interact  with  the  large  screen  display  (LSD)  and  a 
dedicated  small  screen  display  (SSD).  The  C2  task  environment  was  created  using  the 
Distributed  Dynamic  Decision-Making  (DDD)  software  tool.  The  task  involved 
monitoring  a  simulation  of  three  unmanned  air  vehicles  (UAVs),  with  route  and  position 
depicted  on  the  LSD,  and  gauges  representing  the  status  of  specific  aircraft  infonnation 
contained  on  the  SSD. 

4.2. 3.2  Room.  A  10’xl2’  laboratory  at  Wright  Patterson  Air  Force  base  was  used  to 
conduct  the  study.  The  LSD  was  affixed  to  the  back  wall  of  the  laboratory  and  the  SSD 
sat  on  a  table  approximately  6  feet  in  front  of  the  LSD. 

The  SSD  was  a  17”  desktop  monitor  supported  by  a  MIcit  Desktop  running  on  Windows 
XP  Professional  which  displayed  the  lower  monitoring  task  (see  Figure  3). 


Figure  3:  Small  screen  display;  Study  2 


A  wall  mounted  (80”  horizontal,  1280  x  1024  resolution)  projection  screen  served  as  the 
LSD.  Image  was  projected  using  an  InFocus©  projector  from  a  Dell  Dimension  Desktop 
XPS  running  on  Windows  XP  Professional.  Figure  4  depicts  the  LSD. 
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Figure  4:  Large  screen  display;  Study  2 


4.2. 3. 3  Subjective  Measures.  The  NASA-TLX  was  administered  electronically  between 
each  experimental  session  to  collect  participants’  subjective  workload.  A  one-page,  post¬ 
test  survey  was  administered  to  collect  preference  rankings,  perceived  performance,  and 
comfort  level  with  each  communication  technique. 

4.2.4  Procedure 

The  experiment  team  consisted  of  two  persons:  The  Experimenter  and  the  Secondary 
Observer.  The  Experimenter  was  responsible  for  briefing  the  participant,  providing 
instruction  before  each  session,  and  administering  the  NASA  TLX  after  each  condition. 
The  Secondary  Observer’s  role  was  to  receive  communication  from  the  participant 
during  a  session,  and  depending  on  the  condition,  provide  feedback  in  the  form  of 
“reading  back”  the  participant’s  response  and  either:  1)  acknowledging  accuracy,  or  2) 
requesting  a  correction.  Participation  lasted  approximately  90  minutes,  including 
training. 

Participants  completed  a  practice  session  for  each  communication  type.  Within  each 
communication  type,  a  total  of  twenty  communication  events  were  completed.  For  a 
given  participant,  the  first  ten  events  were  readback  conditions,  and  the  remaining  ten 
were  not  (or  vice  versa).  Participants  were  trained  on  the  proper  procedure  for 
responding  to  the  event,  communicating  using  the  applicable  method,  and  (when 
appropriate)  responding  to  readback  from  the  Secondary  Observer.  Order  of 
communication  type  was  controlled  via  Latin  square. 

Participants  conducted  two  concurrent  tasks.  The  first  task  was  to  monitor  the  simulated 
flight  tracks  of  three  UAVs  on  the  large  screen  display;  the  second  task  was  to  monitor 
three  status  windows  on  the  small  screen  display,  which  indicated  the  status  of  three 
critical  UAV  system  parameters  (i.e.,  fuel  remaining,  signal  strength,  engine  temperature) 
of  the  UAVs.  Both  tasks  had  the  potential  for  the  occurrence  of  an  “event.”  A  large 
screen  event,  or  Target  Event,  was  an  encounter  with  a  target  of  opportunity  (artillery, 
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chemBio,  bunker).  A  small  screen  event,  or  Status  Event,  was  an  out-of-limits  reading 
on  one  of  the  key  system  status  parameters. 

Upon  recognizing  the  occurrence  of  an  event,  the  participant  communicated  key, 
predefined  aspects  to  the  experimenter.  The  intent  was  to  require  information  similar  to 
that  on  a  K02.33  CAS  9  Line  form,  which  is  a  standardized  checklist  used  to  document 
and  communicate  key  target  parameters  in  the  battlespace  environment.  For  a  Status 
Event,  these  parameters  were:  Event  Type,  UAV  Name,  Location,  Time  (mm:ss), 
Malfunction,  Level  (High/Low),  and  Corrective  Action.  For  a  Target  Event,  these 
parameters  were:  Event  Type,  UAV  Name,  Target  Name,  Location  X,  Location  Y, 
Altitude,  and  Time  (mm:ss).  A  list  of  these  parameters  was  given  to  the  participant  for 
use  as  a  script  to  follow  for  communicating  to  the  Secondary  Observer. 

During  a  given  session,  participants  communicated  to  the  Secondary  Observer  in  one  of 
three  ways:  spoken  word,  handwritten,  or  electronically  via  chat  software.  Participants 
were  briefed  and  trained  before  each  session  on  the  communication  technique  to  be  used, 
and  were  given  a  chance  to  practice.  Once  comfortable  with  the  assigned  communication 
technique,  the  Experimenter  started  the  trial. 

Spoken  word  required  the  participant  to  orally  communicate  to  the  Secondary  Observer 
each  key  parameter.  In  the  handwritten  condition,  the  participant  completed  a  4”  x  4” 
paper  fonn  containing  each  of  the  parameters  in  the  same  order  as  the  script.  The  chat 
condition  used  Jabber™  as  the  chat  software;  the  participant  typed  the  parameters,  one 
line  per  parameter,  to  the  Secondary  Observer. 

Participants  communicated  a  total  of  twenty  events  using  each  technique.  For  half  of 
each,  the  Secondary  Observer  provided  feedback  to  the  participant,  in  the  fonn  of  a 
“readback”  confirmation  (“[parameter]  OK”)  or  request  to  recheck  for  accuracy 
(“[parameter]  Recheck”). 


4.3  Results 

4.3.1  Response  Time 

A  2-way  repeated-measures  ANOVA  was  conducted  to  examine  the  effect  of  Readback 
and  Communication  Type  on  the  time  taken  to  complete  the  predetennined 
communication  parameters.  The  within- subjects  variables  were  Readback,  with  two 
levels  (Readback  Present  and  Readback  Absent),  and  Communication  Type  (Spoken 
Word,  Handwritten,  and  Chat  Software).  Alpha  level  .05  was  used.  Table  6  presents  the 
ANOVA  Summary  Table  for  this  analysis  while  Table  5  depicts  the  means  and  standard 
deviations. 

As  expected,  a  significant  main  effect  for  Readback  was  found,  F{  1,  9)  =  1 10.503,/?  < 
0.001.  Participants  took  significantly  longer  to  complete  the  task  in  the  Readback 
Present  condition  than  the  Readback  Absent  condition.  A  significant  main  effect  for 
Communication  Type  was  also  found,  F( 2,  18)  =  58.275, p  <  0.001.  Subsequent  pairwise 
comparisons  revealed  that  participants  took  significantly  longer  to  complete  the  task  in 
the  Chat  Software  condition  than  both  the  Handwritten  condition  (p  =  .036)  and  the 
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Spoken  Word  condition  (p  <  .001),  and  took  significantly  longer  to  complete  the 
Handwritten  condition  than  the  Spoken  Word  condition  (p  <  .001). 


Table  5:  Response  time  in  seconds 


Factor 

M 

SD 

Readback  Present 

46.801 

13.954 

Readback  Absent 

27.094 

7.321 

Spoken  Word 

25.528 

6.540 

Handwritten 

40.119 

13.467 

Chat  Software 

45.195 

15.556 

Table  6:  ANOVA  Summary  Table  for  Response  Time 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Medium 

Sphericity  Assumed 

4169637846 

2 

2084818923 

58.275 

.000 

Greenhouse-Geisser 

4169637846 

1.617 

2578032978 

58.275 

.000 

Huynh-Feldt 

4169637846 

1.920 

2171828583 

58.275 

.000 

Lower-bound 

4169637846 

1.000 

4169637846 

58.275 

.000 

Error(Medium) 

Sphericity  Assumed 

643954548 

18 

35775252.68 

Greenhouse-Geisser 

643954548 

14.556 

44238749.09 

Huynh-Feldt 

643954548 

17.279 

37268328.43 

Lower-bound 

643954548 

9.000 

71550505.37 

Condition 

Sphericity  Assumed 

5825214796 

1 

5825214796 

110.503 

.000 

Greenhouse-Geisser 

5825214796 

1.000 

5825214796 

110  503 

.000 

Huynh-Feldt 

5825214796 

1.000 

5825214796 

110  503 

.000 

Lower-bound 

5825214796 

1.000 

5825214796 

110.503 

.000 

Error(Condition) 

Sphericity  Assumed 

474440405 

9 

52715600.60 

Greenhouse-Geisser 

474440405 

9.000 

52715600.60 

Huynh-Feldt 

474440405 

9.000 

52715600.60 

Lower-bound 

474440405 

9.000 

52715600.60 

Medium '•'Condition 

Sphericity  Assumed 

564471115 

2 

282235557.4 

18.752 

.000 

Greenhouse-Geisser 

564471115 

1.287 

438598541.5 

18.752 

.001 

Huynh-Feldt 

564471115 

1.409 

400535416.0 

18.752 

.000 

Lower-bound 

564471115 

1.000 

564471 1 1 4.8 

18.752 

.002 

Error(Medium*Condition) 

Sphericity  Assumed 

270924216 

18 

15051345.32 

Greenhouse-Geisser 

270924216 

11.583 

23390029  82 

Huynh-Feldt 

270924216 

12.684 

21 3601 60.69 

Lower-bound 

270924216 

9.000 

30102690.64 
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Communication  Type 


Figure  5:  Mean  time  to  complete  task;  communication  type  x  readback 

presence 


The  interaction  of  Readback  and  Communication  Type  was  significant,  F(2,  18)  = 
18.752,  p  <  .001,  as  shown  in  Figure  5.  Simple  effects  were  analyzed  to  further  examine 
the  effect  of  Readback  on  each  Communication  Type.  Table  7  presents  the  means  and 
standard  deviations.  Paired  t-Tests  revealed  that  the  effect  was  strongest  for  Spoken 
Word  (t(6)  =  12.131,  p  <  .001),  moderate  for  Flandwritten  (t(6)  =  8.002),  and  weakest  for 
Chat  Software  (t(6)  =  6.406,  p  <  .001). 


Table  7:  Response  time  in  seconds 


Factor 

M 

SD 

Spoken  Word-Readback  Present 

32.080 

3.481 

Spoken  Word-Readback  Absent 

20.994 

3.018 

Handwritten-Readback  Present 

54.643 

8.508 

Handwritten-Readback  Absent 

28.337 

3.495 

Chat  Software-Readback  Present 

57.165 

12.198 

Chat  Software-Readback  Absent 

30.726 

8.047 

4.3.2  Workload 

A  2-way  repeated-measures  ANOVA  was  conducted  to  examine  the  effect  of  Readback 
and  Communication  Type  on  Overall  subjective  workload.  Table  8  presents  the  ANOVA 
summary  table  for  this  analysis  while  Table  9  presents  means  and  standard  deviations  for 
these  groups. 
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There  was  no  significant  main  effect  for  Readback,  F(l,  9)  =  .602,  p  =  .458, 
Communication  Type,  F(2,  18)  =  1.025,  p  =  .379,  or  the  interaction  of  Readback  and 
Communication  Type,  F(2,  18)  =  1.274,  p  =  .304. 


Table  8:  ANOVA  Summary  Table  for  Workload 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Siq. 

Stimulus 

Sphericity  Assumed 

138.827 

3 

46.276 

.405 

.751 

Greenhouse-Geisser 

138.827 

1.625 

85.428 

.405 

.633 

Huynh-Feldt 

138.827 

1.932 

71.845 

.405 

.666 

Lower-bound 

138.827 

1.000 

138.827 

.405 

.540 

Error(Stimuius) 

Sphericity  Assumed 

3086.709 

27 

114.323 

Greenhouse-Geisser 

3086.709 

14.626 

211.048 

Huynh-Feldt 

3086.709 

17.391 

177.490 

Lower-bound 

3086.709 

9.000 

342.968 

Trial 

Sphericity  Assumed 

59.513 

1 

59.513 

1.688 

.226 

Greenhouse-Geisser 

59.513 

1.000 

59.513 

1.688 

.226 

Huynh-Feldt 

59.513 

1.000 

59.513 

1.688 

.226 

Lower-bound 

59.513 

1.000 

59.513 

1.688 

.226 

Error(Trial) 

Sphencity  Assumed 

317.334 

9 

35.259 

Greenhouse-Geisser 

317.334 

9.000 

35.259 

Huynh-Feldt 

317.334 

9.000 

35.259 

Lower-bound 

317.334 

9.000 

35.259 

Stimulus  *  Trial 

Sphencity  Assumed 

40.782 

3 

13.594 

.392 

.760 

Greenhouse-Geisser 

40.782 

2.285 

17.851 

.392 

.707 

Huynh-Feldt 

40.782 

3.000 

13.594 

.392 

.760 

Lower-bound 

40.782 

1.000 

40.782 

.392 

.547 

Error(Stimulus*Trial) 

Sphencity  Assumed 

936.983 

27 

34.703 

Greenhouse-Geisser 

936.983 

20.562 

45.570 

Huynh-Feldt 

936.983 

27.000 

34.703 

Lower-bound 

936.983 

9.000 

104.109 

In  addition  to  Overall  Workload,  2-way  repeated-measure  ANOVA’s  were  conducted  to 
examine  the  effect  of  Readback  and  Communication  Type  on  the  individual  workload 
subscales  (Physical  Demands,  Mental  Demands,  Temporal  Demands,  Own  Perfonnance, 
Effort,  and  Frustration). 


Table  9:  Workload  means  and  standard  deviations 


Factor 

M 

SD 

Readback  Present 

46.045 

17.070 

Readback  Absent 

43.344 

16.737 

Spoken  Word 

42.917 

15.627 

Handwritten 

41.650 

18.396 

Chat  Software 

49.517 

16.059 
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A  significant  main  effect  of  Readback  on  Physical  Demands,  F(l,  9)  =  3.919,  p  =  .046, 
was  observed;  Physical  Demands  were  greater  for  the  Readback  Present  condition  than 
for  the  Readback  Absent  condition.  No  other  effects  were  significant  at  alpha  level  .05. 

4.3.3  Subjective  Rankings 

Participants  were  asked  to  rank  order  the  six  levels  of  Readback  by  Communication  Type 
based  on  subjective  time  performance,  preferred  condition,  and  perceived  value  of 
Readback  for  each  communication  type.  A  within-subjects  comparison  of  these  rank 
orders  was  performed  using  the  Friedman  test. 

4.3.3. 1  Subjective  Performance.  The  Friedman  test  was  significant  (Friedman  X2  = 

18. 1 14,  p  =  .001).  The  six  levels  in  order  of  perceived  perfonnance  are  listed  below,  with 
their  average  placement  (l=most,  6=least): 

•  Spoken  Word-Readback  Absent  (1.9), 

•  Handwritten-Readback  Absent  (2.8), 

•  Spoken  Word-Readback  Present  (3.3), 

•  Chat  Software-Readback  Absent  (3.5), 

•  Handwritten-Readback  Present  (4.5), 

•  Chat  Software-Readback  Present  (5.0), 

Paired  comparisons  revealed  the  following  three  groupings  regarding  perceived 
performance: 

1 .  Spoken  Word-Readback  Absent  ranked  significantly  higher  than 
Chat  Software-Readback  Absent  (p  =  .047),  Handwritten- 
Readback  Present  (p  =  .027),  and  Chat  Software-Readback  Present 

(p  =  .012). 

2.  Handwritten-Readback  Absent  ranked  significantly  higher  than 
Handwritten-Readback  Present  (p  =  .021)  and  Chat  Software- 
Readback  Present  (p  =  .002). 

3.  Chat  Software-Readback  Absent  ranked  significantly  higher  than 
Chat  Software-Readback  Present  (p  =  .025). 

4. 3. 3. 2  Preference.  The  Friedman  test  was  non-significant,  but  approached  significance 
(Friedman  X2  =  9.371,  p  =  .090).  The  six  levels  in  order  of  preference  are  listed  below, 
with  their  average  placement  (l=most,  6=least): 

•  Spoken  Word-Readback  Present  (2.5), 

•  Spoken  Word-Readback  Absent  (2.7), 

•  Handwritten-Readback  Absent  (3.5), 

•  Chat  Software-Readback  Absent  (3.7), 

•  Handwritten-Readback  Present  (3.9),  and 

•  Chat  Software-Readback  Present  (4.7). 

4. 3. 3. 3  Perceived  Value  of  Readback.  The  Friedman  test  was  non-significant  for  rank 
order  based  on  Perceived  Value  of  Readback  (Friedman  X2  =  3.200,  p  =  .222). 
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4.3.4  Errors. 

Errors  were  categorized  as  one  of  two  types:  Critical  and  Non-Critical.  Critical  errors 
were  those  which  would  compromise  the  effectiveness  of  communication.  The  following 
errors  fell  under  this  category:  any  incorrect  information  given,  any  omission  of  checklist 
information,  and  out  of  sequence  checklist  information.  Conversely,  Non-Critical  errors 
were  identified  as  mistakes  made  which  would  not  compromise  information  integrity. 

4.3.4. 1  Critical  Errors.  A  2-way  repeated-measures  ANOVA  was  conducted  to  examine 
the  effect  of  Readback  and  Communication  Type  on  the  number  of  critical  errors 
observed  during  communication.  The  within- subjects  variables  were  Readback,  with  two 
levels  (Readback  Present  and  Readback  Absent),  and  Communication  Type  (Spoken 
Word,  Handwritten,  and  Chat  Software). 

Table  10  provides  the  summary  results  for  the  ANOVA.  No  effect  was  observed  for 
Communication  Type  F(2,  18)  =  1.33,  p  =  .322  or  Readback  F(l,  9)  =  .47,  p  =  .509  on 
critical  errors.  The  interaction  of  Readback  x  Communication  Type  approached  (but  did 
not  reach)  significance  at  the  P=.05  level  [F(l,  9)  =  3.86,  p  =  .067]. 


Table  10:  ANOVA  Summary  Table  for  Critical  Errors 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Medium 

Sphericity  Assumed 

.433 

2 

.217 

1.206 

.322 

Greenhouse-Geisser 

.433 

1.378 

.315 

1.206 

.313 

Huynh-Feldt 

.433 

1.545 

.280 

1.206 

.316 

Lower-bound 

.433 

1.000 

.433 

1.206 

.301 

Error(Medium) 

Sphericity  Assumed 

3.233 

18 

.180 

Greenhouse-Geisser 

3.233 

12.400 

.261 

Huynh-Feldt 

3.233 

13.907 

.232 

Lower-bound 

3.233 

9.000 

.359 

Readback 

Sphericity  Assumed 

.067 

1 

.067 

.474 

.509 

Greenhouse-Geisser 

.067 

1.000 

.067 

.474 

.509 

Huynh-Feldt 

.067 

1.000 

.067 

.474 

.509 

Lower-bound 

.067 

1.000 

.067 

.474 

.509 

Error(Readback) 

Sphericity  Assumed 

1.267 

9 

.141 

Greenhouse-Geisser 

1.267 

9.000 

.141 

Huynh-Feldt 

1.267 

9.000 

.141 

Lower-bound 

1.267 

9.000 

.141 

Medium*  Readback 

Sphericity  Assumed 

.433 

2 

.217 

3.162 

.067 

Greenhouse-Geisser 

.433 

1.931 

.224 

3.162 

.069 

Huynh-Feldt 

.433 

2.000 

.217 

3.162 

.067 

Lower-bound 

.433 

1.000 

.433 

3.162 

.109 

Error(Medium*Readback) 

Sphericity  Assumed 

1.233 

18 

.069 

Greenhouse-Geisser 

1.233 

17.378 

.071 

Huynh-Feldt 

1.233 

18.000 

.069 

Lower-bound 

1.233 

9.000 

.137 

4. 3. 4.2  Non  Critical  Errors.  A  2-way  repeated-measures  ANOVA  was  conducted  to 
examine  the  effect  of  Readback  and  Communication  Type  on  the  number  of  non-critical 
errors  observed  during  communication.  The  within-subjects  variables  were  Readback, 
with  two  levels  (Present  and  Absent),  and  Communication  Type  (Spoken  Word, 
Handwritten,  and  Chat  Software). 
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Table  11:  ANOVA  Summary  Table  for  Non-Critical  Errors 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Medium 

Sphericity  Assumed 

3,900 

2 

1.950 

2.325 

.126 

Greenhouse-Geisser 

3.900 

1.526 

2.555 

2.325 

.143 

Huynh-Feldt 

3.900 

1.775 

2.197 

2.325 

.134 

Lower-bound 

3.900 

1.000 

3.900 

2.325 

.162 

Error(Medium) 

Sphericity  Assumed 

15.100 

18 

.839 

Greenhouse-Geisser 

15.100 

13.738 

1.099 

Huynh-Feldt 

15.100 

15.973 

.945 

Lower-bound 

15.100 

9.000 

1.678 

Readback 

Sphericity  Assumed 

3.267 

1 

3.267 

12.250 

.007 

Greenhouse-Geisser 

3.267 

1.000 

3.267 

12.250 

.007 

Huynh-Feldt 

3.267 

1.000 

3.267 

12.250 

.007 

Lower-bound 

3.267 

1.000 

3.267 

12.250 

.007 

Error(Readback) 

Sphericity  Assumed 

2.400 

9 

.267 

Greenhouse-Geisser 

2.400 

9,000 

.267 

Huynh-Feldt 

2.400 

9.000 

.267 

Lower-bound 

2.400 

9.000 

.267 

Medium*  Readback 

Sphericity  Assumed 

2.033 

2 

1.017 

4.256 

.031 

Greenhouse-Geisser 

2,033 

1.839 

1.106 

4.256 

.035 

Huynh-Feldt 

2.033 

2.000 

1.017 

4.256 

.031 

Lower-bound 

2.033 

1.000 

2.033 

4.256 

.069 

Error(Medium*Readback) 

Sphericity  Assumed 

4.300 

18 

.239 

Greenhouse-Geisser 

4.300 

16.547 

.260 

Huynh-Feldt 

4.300 

18.000 

.239 

Lower-bound 

4.300 

9.000 

.478 

The  main  effect  of  Communication  Type  was  not  significant  F(2,  18)  =  5.49,  p  =  .126. 
The  main  effect  of  Readback  was  significant  F(l,  9)  =  12.25,  p  =  .007;  for  this  main 
effect,  Readback  presence  resulted  in  significantly  fewer  non-critical  errors. 

Figure  6  illustrates  the  significant  interaction  of  Readback  x  Communication  Type  [F(2, 
18)  =  1.02,  p  =  .031].  Analysis  of  simple  effects  revealed  that  Readback  presence 
increased  non-critical  errors  in  the  Chat  condition  [t(9)  =  3.88,  p  =  .004];  however, 
Readback  had  no  effect  on  non-critical  errors  in  the  Flandwritten  condition  [t(9)  =  0.0,  p 
=  1.000]  or  the  Spoken  Word  condition  t(9)  =  2.24,  p  =  .052]. 
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Spoken  Word  Handwritten  Chat  Software 

Co  m  m  u  n  ication  T  ype 

Figure  6:  The  interaction  of  Communication  Type  x  Readback  Presence  on 

non-critical  errors 


4.4  Discussion 

The  results  of  Target  Investigation  2  indicate  that  Spoken  Word  offers  a  perfonnance 
advantage  and  is  preferred  over  Handwritten  and  Chat  formats.  Absence  of  Readback 
offers  faster  response  times  and  was  preferred  over  the  presence  of  Readback,  even  with 
potential  error  correction  offered  by  a  “second  set  of  eyes.”  Finally  the  non-significant 
findings  involving  critical  errors  and  the  significant  interaction  involving  non-critical 
errors  defy  an  expected  speed-accuracy  tradeoff.  These  findings  are  discussed  below. 

The  time  advantage  of  Spoken  Word  over  either  Handwritten  or  Chat  fits  with  the 
concept  that  speaking  is  a  more  universally  understood  medium  of  communication. 
Furthermore,  when  compared  to  chat  and  writing  in  the  present  study,  verbal 
communication  seemed  to  occur  in  a  smoother,  more  familiar  manner.  That  is, 
participants  seemed  to  fall  into  a  pattern  of  speaking  which  they  adopted  throughout  the 
study.  In  contrast,  chat  and  handwritten  required  completion  of  a  line-item  before 
communication  took  place,  which  seemed  to  somewhat  disrupt  the  timing  and  flow  of 
information  exchange,  and  thus  added  to  the  time  advantage  of  speaking  as  a 
communication  medium. 

When  combining  the  above  mentioned  time  results  with  the  results  regarding  preference, 
it  seems  an  easy  decision  that  speaking,  without  the  involvement  of  a  second  person, 
should  be  considered  as  a  primary  communication  technique.  This  is  especially  true  if 
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time  is  a  high  priority.  However,  when  weighing  other  factors  (e.g.,  documentation 
techniques  and  conditions  of  the  overall  work  environment),  voice  may  not  be  the 
preferred  option.  For  example,  the  Targeted  Investigation  2  environment  was  controlled, 
with  no  more  than  three  individuals  (including  the  participant)  present  at  a  given  time. 
Each  piece  of  infonnation  was  conveyed  in  a  controlled  setting,  such  that  infonnation 
could  not  readily  be  lost.  In  an  operational  setting,  with  a  potentially  aurally  saturated 
work  environment,  speaking  performance  may  become  compromised.  Similarly, 
handwriting  may  become  less  legible  during  times  of  high  temporal  demand  or  overall 
workload  levels.  In  these  cases,  mediums  other  than  spoken  word  may  actually  offer  an 
overall  advantage. 

While  the  findings  involving  response  time  seem  fairly  straightforward,  the  findings 
involving  errors  were,  at  least  at  first  glance,  not.  The  intent  of  the  Readback 
requirement  was  to  offer  a  tradeoff  between  speed  and  accuracy,  likely  realizing  a 
decrease  in  errors  while  sacrificing  overall  response  times.  Thus,  the  main  effect  of 
Readback  on  response  time  is  not  surprising  —  more  time  is  generally  needed  if  two 
operators  are  required  to  communicate.  However,  greater  evidence  of  a  speed-accuracy 
tradeoff  was  expected,  on  two  fronts.  First,  the  Secondary  Operator’s  responses  provided 
by  the  Readback  present  condition  was  expected  to  reduce  critical  errors.  This  effect  was 
not  observed.  Second,  in  the  Chat  condition,  the  ability  to  easily  review  one’s  input 
before  submitting  was  anticipated  to  generate  slower,  but  more  accurate  overall 
responses  with  regards  to  both  critical  and  non-critical  errors.  While  the  increase  in  Chat 
time  occurred,  the  reduction  in  critical  errors  did  not  significantly  lessen.  Furthermore, 
for  non-critical  errors,  the  significant  interaction  of  Readback  x  Communication  Type 
illustrated  an  increase  in  the  number  of  non-critical  errors,  when  Readback  was  present. 
Thus,  a  strategy  intended  to  decrease  errors  actually  seemed  to  have  the  opposite  effect. 

To  begin  to  understand  this  seemingly  conflicting  finding,  one  must  investigate  the 
differences  in  nature  between  critical  and  non-critical  errors,  and  the  role  of  the 
Secondary  Operator  in  addressing  them.  To  begin,  the  Readback  present  condition  was 
set  up  to  catch  critical  errors  -  that  is,  errors  involving  mission-essential  data.  As 
mentioned  above,  this  finding  did  not  occur.  On  the  other  hand,  non-critical  errors  were 
not  essential  to  mission  success.  These  errors  involved  the  more  subtle  aspects  of 
accuracy,  such  as  communication  miscues  or  violations  of  protocol.  The  secondary 
Operator  had  little  say  as  to  whether  non-critical  errors  were  made,  and  was  focused  on 
the  correctness  of  the  data.  Thus,  by  nature,  the  Readback  condition  was  more  likely  to 
catch  and  correct  critical  errors. 

To  further  address  the  seemingly  unusual  non-critical  error  findings,  the  characteristics  of 
medium  by  which  communication  took  place  must  also  be  considered.  The  significant 
interaction  of  Readback  x  Communication  Type  illustrates  that,  depending  on  the 
communication  medium,  such  errors  may  increase  as  the  amount  of  communication 
increases.  Specifically,  Chat’s  relatively  recent  emergence  as  a  communication 
technique  may  have  contributed  to  this  finding;  an  overall  protocol  is  not  established,  and 
the  medium  itself  is  susceptible  to  mistakes  such  as  inadvertent  keystrokes  and 
typographical  errors.  These  mistakes  are  easily  made  without  notice.  Moreover,  the  chat 
medium  required  the  operator  to  divide  attention  between  at  least  two  (and  depending  on 
typing  style,  three)  disparately  located  information  sources:  1)  target  information,  2)  chat 
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window,  and  3)  keyboard.  By  contrast,  the  Handwritten  and  Spoken  Word  conditions 
offered  the  advantages  of  familiarity  and,  for  the  Handwritten  condition,  the  option  to 
completely  review  one’s  work  before  submitting.  This  may  have  led  to  identical  non- 
critical  error  findings  in  the  Handwritten  condition.  It  should  be  noted  that  this  strategic 
difference  for  the  Handwritten  protocol  was  seen  by  the  researchers  as  more  practical  in 
implementing  Readback  for  written  form-filling  in  a  C2  environment.  As  such,  overall 
Handwritten  completion  times  may  also  have  been  shortened,  as  compared  to  a  protocol 
more  parallel  to  that  of  Chat  and  Spoken  Word,  which  would  have  required  the  handing 
back  and  forth  of  a  written  checklist  after  completing  each  individual  line  item. 

In  summary,  by  its  nature,  the  presence  of  a  Readback  requirement  likely  generated  the 
potential  for  an  increase  in  non-critical  errors.  Such  an  increase  was  observed  in  the 
current  study,  specifically  when  using  a  relatively  new  technology  -  in  the  case  of  the 
present  study,  chat.  Future  C2  research  should  further  address  these  issues,  with  an 
emphasis  on  what  is  practical  in  the  operational  C2  setting.  Designers,  as  well,  should 
interpret  these  findings  with  the  consideration  that  non-critical  errors  may  be 
manageable,  depending  on  the  communication  technology  and  the  established 
communication  protocol.  The  nature  and  impact  of  such  errors  may  be  an  acceptable 
design  consideration  within  the  larger  picture  of  communication  documentation, 
logistics,  and  work  environment. 


5.0  Conclusion 

The  two  studies  conducted  for  this  project  sought  to  address  situations  in  the  C2  setting 
in  which  an  operator  would  be  required  to  divide  attention.  The  specific  situations 
addressed  were:  1)  how  the  operator’s  attention  could  be  directed  to  relevant  or  critical 
tasking  infonnation;  and,  2)  techniques  to  enable  the  operator  to  communicate 
information  about  those  critical  events  to  a  second  individual.  The  combined  findings 
from  both  studies  depict  a  pattern  of,  which  supports  better  operator  performance  and 
greater  operator  preference,  based  on  increased  familiarity  with  technique  and 
technology.  Such  an  issue  must  be  considered  by  future  researchers,  as  well  as  designers, 
when  considering  whether  to  implement  new  technologies  in  an  environment  with 
established  practices.  The  trade-offs  offered  by  such  implementations  will  be  an 
important  factor;  for  example,  the  ability  to  quickly  and  accurately  archive  chat  sessions 
versus  slower  response  times  and  lower  user  preference  rankings  as  compared  to 
speaking. 

In  conclusion,  three  important  factors  must  be  considered  when  prioritizing  collaborative 
medium:  the  technology,  the  environment,  and  the  user.  Both  Targeted  Investigations 
focused  primarily  on  the  user  and  technology.  Future  research  efforts  could  include 
other,  real-time  scenarios  and  techniques.  Additionally,  further  research  should  take  into 
account  environmental  factors  such  as  noise  levels,  communication  styles,  and  temporal 
issues.  By  including  such  variables  in  future  C2  research,  a  comprehensive  evaluation  of 
optimal  prompting  and  collaboration  techniques  will  be  obtained,  from  both  a  user 
preference  and  performance  standpoint. 
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List  of  Acronyms 


Acronym . Meaning 

AFRL . Air  Force  Research  Laboratory 

ANOVA . Analysis  of  Variance 

C2 . Command  and  Control 

DDD . Distributed  Dynamic  Decision-Making 

DoD . Department  of  Defense 

LOA . List  of  Acronyms 

LSD . Large  Screen  Display 

NASA . National  Aeronautics  and  Space  Administration 

SSD . Small  Screen  Display 

SO . Secondary  Operator 

TLX . Task  Load  Index 

UAV . Uninhabited  Air  Vehicle 

UDRI . University  of  Dayton  Research  Institute 

USAF . United  States  Air  Force 


25 


